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ABSTRACT 

Twenty-two  proposal  competitions  for  government  R&D 
contract,  involving  156  proposal  teams, .^fo  examined  to  de- 
termine the  relative  use  of  three  sources  of  technical  infor- 
mation. The  extent  to  which  each  proposal  team  relied  upon 
literature  search,  the  use  of  staff  specialists  within  the 
l.nhandthe  use  of  outside  sources  of  information  is  related 
to  the  rated  technical  quality  of  its  proposal,  and  to 
other  variables  characterizing  the'  proposal  team  and  Its 
parent  laboratory. 

Twenty-two  percent  of  the  total  time  expended  by  156 
proposal  teams  was  devoted  to  the  seeking  and  gathering  of 
technical  information.  Of  the  three  information  sources 
used, only  one, laboratory  specialists,  appears  to  be  at  all 
directly  related  to  the  technical  quality  of  the  product 
and  this  relation  Is  weak  and  unre I  table. Technical  quality 
Is  Inversely  related  to  the  extent  to  which  the  prDposa'l 
team  relies  upon  Individuals  outside  of  the  laboratory  as 
sources  of  Information. 
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Introduction 

Analysis  of  the  effectiveness  of  the  national  research  and  de- 
velopment effort  directs  attention  to  the  process  of  information 
transfer — the  generation,  storage,  summarization,  and  retrieval  of 
the  ideas  and  data  of  science  and  technology.  The  report  of  the 
President's  Science  Advisory  Committee  on  Science.  Government,  and 
Information  (1963)  stresses  the  essential  point  that  "the  infor- 
mation process  Is  an  integral  part  of  research  and  development. 
Research  and  development  cannot  be  envisaged  without  communication 
of  the  results  of  research  and  development;  moreover,  such  commu- 
nication involves  In  an  intimnte  way  all  segments  of  the  technical 
community,  npt  only  the  documental i sts. 

"We  place  special  stress  upon  what  seems  an  obvious  point  be- 
cause. In  the  early  days  of  science,  the  problem  of  communication 
could  be  managed  casually.   Each  individual  scientist  could  work 
out  his  own  private  communication  system,  suitable  to  his  own  needs, 
and,  since  the  requirements  were  relatively  small,  the  whole  matter 
could  be  treated  rather  incidentally.  But  with  the  growth  of  science 
a  casual  attitude  toward  communication  can  lead  only  to  insufficient 
communication.  Scientists  individually,  technical  societies,  agen- 
cies supporting  research  and  development,  will  have  to  recognize  that 
adequate  communication  no  longer  comes  free.  Communication  cannot  be 
viewed  merely  as  librarians'  worl<,  that  Is,  as  not  really  part  of 
science.  An  appreciable  and  Increasing  fraction  of  science's  resources, 
Including  deeply  motivated  technical  men  as  well  as  money,  will  in- 


evitably  have  to  go  into  handling  the  information  that  science  cre- 
ates" (p. 14). 

The  overriding  questions  of  the  design  and  scale  of  information 
systems  can  only  be  solved,  however,  with  better  knowledge  of  the 
information  needs  of  the  users  and  an  understanding  of  the  role  which 
technical  information  plays  in  research  and  development.   A  first 
step  in  this  direction  is  the  measurement  of  both  the  relative  de- 
pendence upon  and  benefits  derived  from  specific  sources  of  Infor- 
mation.  Information  handling  serves  two  principal  functions:   (I) 
management;  I.e.  direction,  coordination,  reporting,  and  control,  and 
(2)  technical  problem  solution.  This  report  is  concerned  primarily 
with  the  second  function.  Following  Menzel  (1962)  the  study  con- 
'■•iders  the  use  of  and  interaction  among  three  general  sources  of  tech- 
nical information. 

Previous  studies  have  asked  researchers,  by  interview  or  ques- 
tionnaire, what  knowledge  they  needed,  what  sources  they  made  use  of, 
and  what  function  the  information  served  (Brownson,  I960).  Those 
studies  have  been  analyzed  and  compared  In  a  monograph  prepared  for 
the  National  Science  Foundation  by  the  Bureau  of  Applied  Social  Re- 
-^^nnrch  of  Columbia  University  (1960).  Other  work  has  dealt  with  the 
reading  behavior  of  scientists  (Hensley,  1962;  Scott,  1962),  and  the 
flow  and  use  of  information  and  ideas  in  university  and  industrial  re- 
search (Bureau,  1958;  Rubenstein  and  Avery,  1959;  American  Psychol- 
ogical Association,  1963).   It  should  be  noted  that  the  judgment  of 
the  researcher  has  been  the  only  criterion  of  the  value  of  the  infor- 
mation.  In  this  study,  an  objective,  external  evaluation  of  so- 


lutions  Is  related  to  the  information  source  utilized. 

COMPARISON  OF  INFORMATION  SOURCES 
Research  on  research  Is  intrinsically  difficult  because  the 
outcomes  of  different  projects  cannot  be  directly  compared.  The 
opportunity  to  obtain  replication  of  attempted  solutions  to  the 
same  research  problem  is  rare.   Seldom  do  two  or  more  groups  un- 
dertake to  solve  the  same  problem  at  the  same  time  (same  state  of 
knowledge).  This  is  inevitable  because  any  research  study  attempts 
to  perform  a  unique  task.   If  the  problem  has  previously  been 
solved  it  Is,  by  definition,  not  a  research  problem.  Yet  in  order 
to  relate  exposure  to  information  channels  with  performance,  an 
instance  must  be  sought  in  which  the  same  problem  Is  attempted  by 
two  or  more  research  groups. 

The  present  study  utilizes  a  relatively  unique  situation  in 
which  simultaneous  parallel  research  activity  exists,  that  of 
the  government-sponsored  R&D  proposal  competition. 

In  contracting  for  research  and  development,  a  government 
agency  often  solicits  proposals  from  a  number  of  firms  having  pre- 
vious experience  or  Interest  in  the  relevant  field  of  endeavor.. 
The  technical  staff  of  the  agency  draws  up  a  work  statement  de- 
scribing the  mission  the  system  is  to  perform  and  setting  forth 
certain  criteria  to  which  the  design  must  conform.  This  work 
statement  Is  incorporated  in  a  formal  Request  for  Proposal  (RFP) 
which  Is  sent  to  the  chosen  firms.  A  period  of  30  days  to  six 


weeks  is  usually  allowed  for  the  firms  to  prepare  their  responses. 
A  response  will  generally  comprise  three  sections:  technical, 
management  nnd  cost.  This  study  is  concerned  with  the  information 
sources  employed  during  the  preparation  of  the  technical  portion 
of  the  proposal  and  with  the  proposal's  subsequent  evaluation. 
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METHOD 

Data  Col  lection 

I 
Twelve  contracts  were  selected  from  the  files  of  the  United 

States  Air  Force's  Electronic  Systems  Division.  The  Air  Force 
Cambridge  Research  Laboratory  was  the  agency  responsible  for  tech- 
nical evaluation  of  the  proposals  for  these  contracts.  Ten  con- 
tracts were  selected  from  the  files  of  the  National  Aeronautics 
and  Space  Administration's  Marshall  Space  Flight  Center  (MSFC); 
the  proposals  for  these  contracts  were  evaluated  by  the  technical 
personnel  of  MSFC.  The  ten  USAF  contracts  ranged  in  value  from 
$11,000  to  $556,000;  the  NASA  contracts  from  $30,000  to  $169,000. 
At  both  centers  evaluations  are  performed  by  specialists  who  eval- 
uate technical  quality  of  the  proposals  separate  from  and  inde- 
pendent of  the  cost  for  which  each  laboratory  proposes  to  perform 
the  job.  Contract  awards  are  made  by  others  after  cbnsidering 
both  technical  quality  and  cost. 

Questionnaires  were  sent  to  the  managers  of  all  198  proposal 
teams  competing  for  the  contracts  to  elicit  Information  on  the 
characteristics  of  their  proposal  effort.  Satisfactory  returns 
were  obtained  from  156  proposal  managers  in  112  firms.  The  number 
of  replies  for  the  individual  competitions  ranges  from  four  to 
twelve  with  a  median  of  seven. 


I 

The  twelve  contracts  Include  two  which  were  studied  and  reported  In  a 
previous  paper  (Allen  and  Marquis,  1963).   Interviews  were  conducted 
with  most  of  the  21  proposal  managers  In  the  preliminary  study  In  order 
to  test  and  refine  the  questionnaire. 


Data  Reduction  and  Analysis 

The  questionnaire  asks  the  extent  to  which  the  proposal  team 
members  utilized  each  of  three  sources  of  Information:   literature 
search,  specialists  within  the  company  (but  not  assigned  to  the  team), 
and  outside  sources  (consultants,  potential  vendors,  university  pro- 
fessors, etc.).  Additional  data  are  gathered  on  the  characteristics 
of  the  proposal  team,  its  individual  members,  and  the  laboratory 
(Table  I).  For  correlations  with  technical  rank,  a  non-parametric 
measure,  the  Kendall  tau  coefficient  (.'),    (Kendall,  1962),  has  been 
chosen  since  the  data  on  technical  evaluations  are  in  rank-order 
form.  This  limitation  does  not  hold  true  for  the  other  variables, 
and  Pearson  product  moment  correlations  (.'^)   are  computed  for  their 
interrelations.  For  each  proposal  competition  there  are  4  corre- 
lations with  the  criterion  of  technical  quality,  and  37  jntercor- 
relations  among  the  remaining  variables. 

In  order  to  aggregate  the  data  from  22  sets  of  correlations 
into  composite  scores  for  each  variable  a  weighted  average  cor- 
relation is  employed  (Moroncy,  1956),  This  method  weights  individual 
correlations  by  using  the  Fisher  z  transform  and  a  measure  of  sample 

size  for  each  correlation  (n.  -3,  where  n.  is  the  number  of  obser- 

2 
vat  ions  in  each  correlation). 

In  a  survey  of  this  sort,  there  is  always  the  danger  that  mis- 
sing data  will  bias  the  results.  For  example,  if  the  42  proposal 
managers  who  failed  to  return  questionnaires  were  those  who  had  man- 
aged very  unsuccessful  proposals  (near  the  lower  extreme  technical 
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For  a  description  of  this  technique  and  derivation  of  a  method  for  test- 
ing statistical  significance  of  weighted  average  correlations,  see  Allen 
and  Marquis,  1964. 


TABLE  I 
VARIABLES  MEASURED  IN  THE  STUDY 


Category 


Variable  Number 


Description 


Information  sources 


Other  characteristics  of  the 
proposal  effort 


Characteristics  of  the  pro- 
pose I  team 


Characteristics  of  the  parent 
laboratory 


Technical  quality 


Total  time  spent  in  consulting 
with  information  sources(2+3+4) 

Time  spent  in  literature  search 

Time  spent  consulting  with  spe- 
cialists within  the  laboratory 

Time  spent  consulting  with 
sources  of  Information  outside 
of  the  laboratory 


Level  of  effort 

Time  spent  in  analytic  design 

Time  spent  in  breadboard ing  & 
other  benchwork 


Proposal  team  si  ze 

Level  of  education  and  expe- 
rience of  the  proposal  team 


Size  of  technical  staff 

Ratio  of  technical  staff  to  to- 
tal employment 
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ranking)  the  sample  and  results  would  be  biased  toward  more  success- 
ful proposal  teams.  To  test  this  possibility,  the  frequency  dis- 
tribution with  respect  to  technical  rank,  of  the  completed  question- 
naires was  tested  against  the  frequency  distribution  of  ranks  in 
the  total  population.  A  Kolmogorov-SmI rnov  One  Sample  Test  reveals 
no  significant  difference  between  the  two  distributions. 

RESULTS 
Total  Time  Spent  in  Consulting  with  Information  Sources 

Research  and  development  may  be  characterized  as  an  iterative 
problem  solving  process,  with  proposal  preparation  representing,  if 
not  the  first,  then  one  or  several  of  the  early  iterations.   Each 
iteration  furthermore  involves  a  hierarchical  ordering  of  design  de- 
cisions. Each  decision,  in  turn,  is  reached  on  the  basis  of  some 
quantity  of  Information  describing  desired  performance  and  the  en- 
vironmental and  technical  constraints.  This  paper  concerns  itself 
with  the  sources  of  information  employed  in  coming  to  design  de- 
cisions during  proposal  preparation. 

The  parameter,  "total  time  spent  in  consulting  with  Information 
sources",  Is  defined  as  the  summation  of  the  time  spent  In  literature 
search,  and  in  consulting  with  technical  specialists  both  within  and 
outside  of  the  company.  Twenty-two  percent  of  the  total  time  expended 
by  the  156  proposal  teams  was  devoted  to  the  seeking  and  gathering  of 
Information  from  these  three  sources.  Total  time  spent  consulting  in- 
formation sources  is  unrelated  to  the  rated  technical  quality  of  the 
proposal  (Table  II).  This  is  true  even  when  highly  Intercorrelated  var- 


iables  such  as,  the  level  of  effort  in  preparing  the  proposal;  time 
spent  In  analytic  design;  proposal  team  size;  or  ratio  of  technical 
staff  to  total  employment  are  taken  into  account.   It  will  be  seen 
that  this  is  at  least  partially  a  result  of  relations  between  technical 
quality  and  individual  sources  of  information  which  oppose  and  nullify 
one  another.   To  overcome  this  deficiency,  multiple  correlations 
relating  each  variable  to  variables  one,  two  and  three  (R.  .  2  3)  are 
computed  and  averaged  over  the  22  proposal  competitions.   In  case  of  tech- 
nical rank,  such  a  procedure  is  not  strictly  legitimate,  and  this  is 
the  principal  reason  for  resorting  to  a  derived  summation  variable. 
In  addition,  of  course,  the  summation  variable  provides  an  indication 
of  direction  and  allows  the  computation  of  partial  correlations.   For 
variables  other  than  technical  rank,  both  the  multiple  correlation 
coefficient  and  the  correlation  coefficients  for  the  summation  vari- 
able are  shown  In  Table  II. 

Since  the  investigator  was  still  interested  in  gaining  at  least 
an  estimate  of  the  total  impact  of  the  three  Information  sources 
upon  technical  quality  of  the  proposal,  the  Invalid  assumption  that 
technical  ranks  form  an  interval  scale  was  made  and  multiple  corre- 
lations performed.  Given  this  assumption,  the  three  Information 
sources  are  found  to  account,  on  the  average,  for  50^  of  the  variation 
in  technical  rank. 


3 
The  nullification  effect  is  peculiar  to  the  way  in  which  the  variable 
is  defined.   Since  it  is  the  summation  of  the  values  of  three  other  vari- 
ables, if  one  of  the  three  variables  is  directly  related  to  a  fourth,  and 
one  or  both  of  the  remaining  two  are  inversely  related  to  the  fourth  vari- 
able, then  the  correlation  of  the  summation  variable  with  the  fourth  vari- 
wi I  I  be  somewhere  between  the  extremes.   In  other  words,  if,  for  example, 
r,  ^  =  0.50,  r_  ^  =  -0.50  and  r    =  0,  then  -0.50  <  r^  ^  <0.50,  where 

1,4  ^/^  -5»*  5,4 

variable  five  Is  the  surrimation  of  variables  one,  two  and  three. 


TAOLE  II 
USE  OF  ALL  INFORMATION  SOURCES 


Moan  time  consulting  all  information  sourcen:  55  man-hiours 

Median  time  consulting  all  information  sources:  25  man-hours 

Range:  0-518  man-hours 

Number  of  proposal  teams:  156 

Number  of  proposal  competitions; ^ 22 


Relation  of  total  time  spent  cohsulting  with 
information  sources  (I)  to: 


Average  correlation 
coefficient  (for  the 
summation) 


Average  multiple 
correlation  co- 
efficient Rj .2  3  4 


0.  Technical  quality  of  the  proposal 

5.  Level  of  effort 

6.  Time  spent  in  analytic  design 

7.  Time  spent  In  breadboard  I ng  and  other 
benchwork 


8.  Proposal  team  size 

9.  Average  level  of  education  of  proposal 
team  *** 

10.  Size  of  technical  staff 

11.  Rotio  of  technical  staff  to  total 
emp loyment 

Partial  Correlations: 

Technical  quality  with  level  of  effort 
constant  (tj  ^  ^) 

Technical  quality  with  time  in  analytic 
design  constant  (t|  q  6^ 

Technical  quality  with  proposal  team 
size  constant  (t.  q  g) 

Technical  quality  with  ratio  of  tech- 
nical staff  to  total  employment  con- 
stant (t|^o. I  I  J 

Time  in  analytic  design  with  level  of 
effort  constant  (r.  ^  c) 

Level  of  effort  with  proposal  team 
size  constant  (r|  5  g) 

Proposal  team  size  with  level  of  ef- 
fort constant  (r,  _  j.) 

I  .0,  5 


-0.02 
0.50** 
0.40** 
0.26* 

0.35* 
-0.07 

-0.06 
-0.36** 

-0.03 

-0.05 

-0.05 

0.05 

0.23 
0,71 
0.05 


0.79 
0.83 
0.85 

0.71 


0.73 


*  significant  at  0.05  level,  **   significant  at  0.001  level 
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Time  spent  gathering  information  is  strongly  related  to  the  time 
spent  In  processing  It  (analytic  design),  even  when  level  of  effort 
Is  held  constant  (Table  II).  Partial  correlations  show  that  the  re- 
lation between  use  of  information  sources  and  proposal  team  size  Is 
simply  a  result  of  the  fact  that  larger  proposal  teams  engage  in  more 
total  activity  and  use  of  information  sources  increases  accordingly. 
This  can  be  seen  quite  clearly,  since  the  correlation  between  use  of 
Information  sources  and  proposal  team  size  disappears  when  controlled 
for  level  of  effort  (r,  35=  0.05),  but  the  converse  grows  stronger 

The  amount  of  time  spent  consulting  information  sources  Is  in- 
versely related  to  the  ratio  of  technical  to  total  employment  In  the 
lab.  This  Indicates  the  possibility  that  laboratories  with  a  higher 
proportion  of  technical-professional  personnel  are  able  to  assign 
engineers,  who  are  more  experienced  or  more  competent  In  the  requisite 
technical  area,  and  have  to  rely  less  upon  external  sources  of  tech- 
nical Information.  The  absence  of  a  correlation  in  the  case  of  edu- 
cational level,  can  be  Interpreted  in  this  line  of  thlhklng  to  mean 
simply  that  this  measure  is  Inadequate  to  Identify  competence  In  par- 
ticular technical  areas. 

l^lterature  Search 

Published  literature  has  traditionally  been  the  vehicle  for  the  com- 


*♦*  arrived  at  by  arranging  educational  attainment  levels  In  the  following 
ordinal  scale,  and  using  rank  order  correlations; 

1.  Ph.D.  engineering  or  science 

2.  M.S.  engineering  or  science 

3.  B.S.  engineering  or  science 

4.  No  college  degree,  but  engineering  Job  classification 

5.  B.B.A.  or  B.S.  In  Business  Administration 
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munication  of  scientific  and  technological  findings.  Tlie  proposal 
writer  is  In  a  position  where  he  must  be  conversant  with  the  state- 
of-the-art  In  many  areas;  he  must  search  out  cues  from  all  fields 
of  science  and  technology.   As  a  result,  those  engineers  who  are 
primarily  engaged  In  new  business  areas  must  work  continuously  to 
keep  abreast  of  developments. 

In  asking  proposal  managers  to  estimate  the  number  of  man-hours 
spent  by  their  teams  in  literature  search,  we  find  this  to  be  one 
of  the  most  difficult  of  our  questions  to  answer.   In  many  cases, 
the  respondent  will  make  an  estimate  but  then  noie  that  since  this 
Is  a  continuing  activity  it  is  very  difficult  to  accurately  apportion 
it  among  the  various  projects  and  proposal  preparations  which  might 
be  underway. 

Nevertheless,  for  the  sample,  literature  search  is  the  most 
heavily  employed  means  of  seeking  information.   It  represents  57$   of 
the  Information  gathering  time  and  11^  of  the  total  time  in  proposal 
preparation.   No  relation  is  found  in  aggregate  between  literature 
search  time  and  rated  technical  quality  (Table  III).  Of  the  corre- 
lations on  Individual  competitions,  only  two  are  statistically  signi- 
ficant at  the  0.05  level.  One  of  these  was  positive  {,-\^   q  =  0.75) 
and  the  other  negative  (t2  ^  =  -0.54).  The  absence  of  a  relation  Is 
not  changed  by  accounting  for  total  time  spent  In  proposal  preparation; 
time  spent  In  consulting  with  laboratory  specialists;  or  time  spent 
consulting  with  outside  sources  of  Information. 

Literature  search  is  not  used  Id  the  exclusion  of  either  of  the 
other  Information  sources  considered.  Those  teams  which  rely  more  heavily 


TAr3LE  I  I  I 
USE  OF  LITERATURE  SEARCH 


Mean  time  in  literature  search:  28.4  man-hours 

Median  time  In  literature  search:  15  man-hours 

Range:  0-450  man-hours 

Number  of  proposal  teams:  156 

Number  of  proposal  competitions:  22 


Relation  of  time  spent  In  literature  search  (2)  to: 


Average  correlation 
coefficient 


0. 

Technical  quality  of  the  proposal 

i   . 

3. 

Time  spent  consulting  with  specialists  within  the  laboratory 

0.A5** 

4. 

Time  spent  consulting  with  outside  sources 

0.-59** 

5. 

Level  of  effort 

0.48** 

6. 

Time  spent  In  analytic  design 

0.18 

7. 

Time  spent  In  breadboard Ing  and  other  benchwork 

0.23* 

8. 

Proposal  team  size 

0.25* 

9. 

Average  level  of  education  of  the  proposal  team'''^** 

-0.13 

10. 

Size  of  technical  staff 

-0.12 

11. 

Ratio  of  technical  staff  to  total  employment 

-0.21* 

Partial  correlations: 

Technical  quality  (with  time  spent  ^consulting  with 
laboratory  specialists  constant)  (1^2,0.2) 

0.01 

Technical  quality  with  time  ^pent  consulting  with  out- 
side sources  constant  {^2   0.3^ 

0.05 

Technical  qualltyCwith  level  of  effort  constant) 
(+2,0.5> 

0.04 

Technical  quality  with  ratio  of  technical  staff  to 
to  total  employment  constant  (t      ) 

Ratio  of  technical  staff  to  total  employment  with 
level  of  effort  constant  (r2  11,5) 

0.08 
-0.13 

*  significant  at  0.05  level 
**     significant  at  0.001  level 
***  Rank  order  correlations.  Refer  to  Table  II  for  the  method  of  ranking 
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upon  the  literature  use  at  lea^t  one  of  the  other  sources  to  a  greater 
extent  as  well.  Tl?  nature  of  the  contract  belnp  sought  affects  the 
choice  between  the  o'-her  two  sources.  On  proposals  for  contractr^  in 
excess  of  $80,000  (meolsn  price),  teems  rely  more  on  the  literature 
also  used  laboratory  sf^ecialists  more;  on  proposals  for  contracts 
which  were  less  than  $80,000  in  value,  teams  using  the  literature  more 
consulted  outside  sources  more.  This  may  be  attributed  to  the  fact 
that  the  larger  contracts  attract  larger  laboratories  and  these  have 
the  staff  available  for  technical  consultation.  Smaller  firms  seek 
information  through  the  literature  and  through  other  sources  outside 
of  the  lab. 

Larger  laboratories  and  those  with  a  higher  ratio  of  engineers 
and  scientists  spend  less  time  with  the  literature.  Presumably  these 
labs  are  able  to  fonn  proposal  teams  with  a  higher  level  of  education 
and  experience  and  their  members  are  either  able  to  rely  more  upon 
Informatbn  gained  through  their  education  and  experience  or  are  more 
efficient  in  their  use  of  the  literature.  A  weak  Inverse  relation  be- 
tween literature  search  time  and  level  of  education  .-^nds  to  support" 
this  possibi llty. 

Figure  I  Illustrates  this  situation.  Each  point  in  this  figure 
represents  the  average  value  across  the  22  proposal  competitions  of 
the  normalized  time  spent  in  literature  search  by  proposal  teams  placing 
in  a  given  rank.  Actual  man-hours  reported  are  normalized  relative  to 
the  mean  value  for  a  competition  in  order  to  afford  comparability  among 
the  22  competitions.   It  Is  readily  apparent  that  none  of  the  proposal 
teams  deviated  significantly  from  the  mean  time  (1.0  on  the  abscissa) 
for  Its  particular  competition. 
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Time  Spent  with  Laboratory  Specialists 

Allen  and  Marquis  (1964)  have  discussed  the  economics  of  the  as- 
signment of  personnel  to  the  proposal  team.  They  show  that  very  often 
it  is  undesirable  or  impractical  to  assign  the  best  talent  in  a  lab- 
oratory toa  proposal  team  on  even  a  part-time  basis.  There  are  many 
reasons  for  this.  The  expected  gain  from  tbe  proposal  may  not  ap- 
proach the  opportunity  cost  of  removing  these  men  from  other  projects; 
the  portion  of  the  work  actually  requiring  their  talent  may  be  quite 
small;  or  it  may  be  more  desirable  to  force  others  to  increase  their 
knowledge  in  a  particular  area  and  the  writing  of  a  proposal  (since 
it  will  be  evaluated)  provides  the  necessary  motivation  to  accomplish 
this.   At  any  rate,  the  best  men  are  often  not  assigned  to  the  pro- 
posal team.  They  are,  however,  generally  available  to  the  proposal 
team  members  for  advice  and  consultation.   Such  a  compromise  is  com- 
patible with  the  above  objectives  and  may  still  be  an  efficient 
means  of  bringing  the  necessary  talent  to  bear  upon  the  proposal. 

To  determine  the  extent  to  which  this  practice  is  followed,  the 
survey  asked  proposal  managers  to  estimate  the  number  of  man-hours 
which  their  men  spent  in  consulting  with  specialists  within  the  lab- 
oratory who  were  not  assigned  to  the  proposal  team.  Use  of  specialists 
within  the  laboratory  represents  5\%   of  the  information-gathering  ef- 
fort on  the  part  of  the  proposal  teams  and  1%   of  their  total  effort. 

Results  of  the  pilot  study  of  two  proposal  competitions  (Allen 
and  Marquis,  1963)  suggested  the  hypothesis  that  technical  quality  is 
directly  related  to  the  extent  to  which  functional  specially  within 
the  laboratory  are  consulted  on  technical  matters.   With  the  additional 
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TABLL  IV 
USE  OF  SPECIALISTS  WITHIN  THE  LABORATORY 


Mean  time  in  consulting  with  laboratory  specialists: 

Median  time  in  consulting  with  laboratory  specialists: 

Range: 

Number  of  proposal  teams: 

Number  of  proposal  competitions: 


17.2  man-hours 
8  man-hours 
0-200  man-hours 
156 
22 


Relation  of  time  spent  consulting  with  laboratory 
special ists  (3)  to: 


0.  Technical  quality  of  the  proposal 

2.  Time  spent  in  literature  search 

4.  Time  spent  consulting  with  outside  sources 

5.  Level  of  effort 

6.  Time  spent  in  analytic  design 

7.  Time  spent  in  breadboard ing  and  other  benchwork 

8.  Proposal  team  size 

9.  Average  educational  level  of  the  proposal  team*^<^«- 

10.  Size  of  technical  staff 

11.  Ratio  of  technical  staff  to  total  employment 

Partial  correlations: 

Technical  quality  with  level  of  effort  constant 

<+3,0.5> 

Technical  quality^with  time  spent  in  literature 

search  constant  (t3^o.|) 

Tnchnical  quality  with  time^  spent  consulting  with 
outside  sources  constant  (T3  o.3^ 

Technical  quality^with  time  spent  in  analytic 
design  constant  (t3^o.6) 

Technical  quality  with  proposal  team  size 
constant  (t,  q  g) 

Technical  quality  with  ratio  of^  technical  staff 
to  total  employment  constant  (t3^o.l|) 

Level  of  effort  with  propcsal  team  size  constant 

(^3  5.8) 

Proposal  team  size  with  level  of  effort  constant 

^^3,6.5^ 


Average  correlation 
coefficient 


0.12 

0.43** 

0.09 

0.39*» 

0.56** 

0.23* 

0.44** 

-0.12 
0.25* 

-0.30* 

0.14 

0.12 

0.13 

0.08 

0.03 

0.14 

0.36 
0.25 


significant  at  0.05  level,  **   significant  at  0.001  level 
refer  to  Table  II  for  the  method  of  ranking. 


***  Rank  order  correlations, 
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data  now  available,  the  relation  is  in  the  predicted  direction  (Table  IV) 
but  is  weal<er  than  expected,  although  statistically  significant  at  the  0.05 
level  (one-tailed).  The  plot  of  means  of  normalized  values  for  each 
rank  in  figure  2  Illustrates  the  rather  weak  relation  obtained. 

Reliance  upon  laboratory  specialists  is  strongly  related  to  the  size 
of  the  proposal  team:   the  larger  the  team,  the  greater  the  contact  with 
specialists  who  are  not  on  the  team.  The  Increased  use  of  laboratory 
specialists  by  larger  proposal  teams  can  of  course  be  attributed  to  the 
fact  that  larger  proposal  teams  generally  spend  more  time  in  all  ac- 
tivities. The  relation,  however,  remains  strong  even  when  controlled 
for  level  of  effort,  indicating  that  the  increased  interaction  with 
other  members  of  the  organization  is  a  function  of  the  size  of  the 
proposal  team 'per  se  as  well  as  being  a  result  of  their  increased  general 
effort. 

Thus  we  see  that  increasing  the  size  of  the  proposal  team  increases 
its  ability  to  perform  information  gathering  functions.   As  new  members 
Join  the  team,  they  bring  with  them  potential  links  to  distant  parts 
of  the  organization  which  may  provide  information  of  value  to  the  pro- 
posal.  Such  potential  communication  links  develop  as  a  result  of  an 
individual's  experience  within  the  organization  over  a  number  of  years. 
Increasing  the  size  of  a  proposal  team  and  bringing  In  people  from  di- 
verse parts  of  the  lab  may  well  be  an  effective  way  to  make  more  complete 
use  of  the  broad  range  of  competences  available,  and  to  stimulate  greater 
interaction  and  communication  among  these  organizational  parts.   Pelz 
(1956)  has  shown  scientific  performance  to  be  related  to  the  frequency 
of  contact  with  "scientific  colleagues  who  on  the  average  have  been  em- 
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ployed  in  scientific  settings  different  from  one's  own,  whto  stress 
values  different  from  one's  own,  and  wlio  tend  to  worl<  in  scientific 
fields  different  from  one's  own."  Tliis  may  indicate  a  way  to  improve 
the  performance  of  a  research  organization  by  increasing  the  pro- 
bability of  contact  among  its  researchers  who  are  working  in  remotely 
related  areas. 

The  larger  laboratories  (those  having  both  large  technical  staffs 
and  a  low  ratio  of  tech  lical  staff  to  total  employment)  are  the  ones 
which  principally  resort  to  this  device.   In  many  cases,  these  labs 
use  men  having  less  education  in  their  proposal  teams,  but  compensate 
by  increasing  the  size  of  the  team  and  thereby  its  propensity  to  In- 
teract with  and  gather  Information  from  other  segments  of  the  or- 
ganization. This  practice  not  only  leaves  the  technical  specialists 
free  to  work  on  other  projects  without  seriously  injuring  the  pro- 
posal's quality,  but  It  has  such  important  side  benefits  as  educating 
the  people  who  are  assignod  to  the  team,  and  maintaining,  through 
exercise,  the  communication  lines  among  far-flung  parts  of  the  or- 
ganization. Of  course,  such  a  practice  can  be  carried  too  far.   If 
too  many  proposal  team  members  begin  wandering  about  the  organization 
asking  others  to  do  bits  and  pieces  of  their  work  for  them,  the  point 
can  easily  be  reached  where  the  staff  specialists  will  have  little 
time  for  their  normal  assignments.   Since  the  reward  structure  in  R  &  D 
labs  is  normally  based  on  an  evaluation  of  the  contributions  which  a 
man  makes  within  his  own  group  and  disregards  any  contribution  which 
he  might  make  as  a  consultant  to  another  group,  a  practice  such  as  this 
is  bound  to  arouse  resentment  on  the  part  of  the  consultants.  This  can, 
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of  course,  be  countered  by  moderation  in  practice  and  perhaps  by 
recognizing  the  value  of  certain  of  the  specialists  as  consultants, 
and  modifying  the  reward  structure  accordingly. 

Time  Spent  with  Outside  Sources  of  Information 

Many  writers  on  the  subject  of  proposal  preparation  recommend 
the  use  of  outside  consultants  as  a  possible  substitute  for  ex- 
pertise not  available  within  the  laboratory  (cf.  Bjorl<sten,  1965; 
Karger  and  Murdicl<,  1963).  Conversations  with  both  proposal  managers 
and  professional  consultants  reveal  this  to  be  a  rather  popular  de- 
vice for  compensating  in  specific  critical  areas  of  technology.   In 
spite  of  Its  general  acceptance,  there  is  some  evidence  of  dis- 
satisfaction wjth  the  system  as  presently  used.  Proposal  managers 
often  complain  that  they  don't  get  their  money's  worth  from  con- 
sultants, or  that  even  Informal,  unpaid  consulting  is  not  worth 
the  time  required  (opportunity  cost);  consultants  on  the  other  hand 
reply  that  they  are  brought  on  board  too  late  and  then,  only  when 
the  proposal  team  has  proceeded  too  far  with  the  remainder  of  the 
design  In  Ignorance  of  the  problem  area.  The  data  support  these 
complaints  to  the  extent  of  confirming  the  existence  of  some  serious 
difficulty  in  the  relationship  between  the  proposal  team  and  In- 
formation sources  outside  of  the  company. 

Outside  sources  were  defined  to  the  respondents  In  a  very  broad 
sense  as  any  individual  or  group  outside  of  the  company  who  made  a  tech- 
nical contribution  to  the  problem.  This  includes  consultation  on  any 
basis:   paid  or  unpaid,  formal  or  informal,  and  would  include  colleagues 
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In  professional  societies,  potential  vendors,  or  neighbors,  as  well 
as  professional  consultants.  Time  spent  consulting  with  outside 
sources  represents  15.9^  of  the  total  information  gathering  time 
and  3.5J5  of  the  proposal  preparation  time.   In  terms  of  time  ex- 
pended then  It  is  a  relatively  minor  component  of  the  proposal  effort; 
in  terms  of  providing  l<ey  problem  solutibns  it  can,  of  course,  be 
critical.  Outside  sources  were  employed  by  more  than  one  laboratory 
In  15  of  the  22  proposal  competitions.  The  correlation  analysis 
which  follows  is  restricted  to  these  15  competitions. 

Teams  which  rely  more  heavily  upon  outside  sources  produce  poorer 
quality  solutions  (Table  V).  Fourteen  of  the  fifteen  correlations 
for  Individual  competitions  are  negative  and  range  from  -0.04  to 
-0.74.  Five  6i   the  fourteen  individual  correlations  are  statistically 
significant  at  the  0.05  level.  The  single  positive  correlation 
(ty  0  =  0.42,  not  significant)  involves  an  experimental  investigation 
of  plastic  film  balloon  materials.  Figure  3  illustrates  the  inverse 
relation  by  plotting  an  average  of  the  normalized  values  for  each 
technical  ranl<  across  the  15  competitions.  While  the  indications 
are  indeed  strong  that  there  is  an  inverse  relation  between  outside 
consulting  and  rated  quality,  evidence  of  this  sort  should  not  be  con- 
strued to  Imply  causality.  Those  teams  which  rely  upon  outside  help 
possess  other  characteristics  which  more  likely  are  the  actual  cause  of 
the  poor  performance.  The  most  plausible  of  these  is  simply  the  lacl< 
of  the  required  technical  competence  within  the  lab.  This  Is  supported 
by  strong  Inverse  relations  between  the  use  of  outside  sources  and  both 
the  size  of  the  lab's  technical  staff  and  its  ratio  to  the  lab's  total 
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employment.  Partial  ling  out  these  two  variables  results  in  a  positive 
correlation  between  tiie  use  of  outside  sources  and  technical  quality.  , 
Laboratories  which  do  not  have  the  necessary  technical  manpower  re- 
sources attempt  unsuccessfully  to  substitute  through  reliance  upon  out- 
side technical  personnel.  The  failure  Is  probably  due  to  a  problem  of 
communication  with  outsiders.  One  of  the  more  important  components  of 
a  technical  proposal  is  the  demonstration  of  an  Intimate  understanding 
of  the  problem.   Evaluators  are  not  looking  for  final  solutions;  what 
they  are  lool<lng  for  is  a  good  first  iteration  which  demonstrates  thorough 
understanding  of  the  nature  of  the  problem  being  faced  and  of  the 
customer's  requirements.  This  minimizes  the  risl<  of  engaging  a  laboratory 
that  will  go  off  In  wrong  directions  and  make  grossly  inaccurate  pre- 
dictions of  the  capability  or  feasibility  of  approaches.   Even  when 
the  proposal  team  members  themselves  have  a  thorough  understanding  of 
the  problem,  it  can  be  very  difficult  and  time  consuming  to  raise  an 
outsider  to  this  state  of  l<nowledge. 

Every  R&D  laboratory  has  its  own  way  of  attacl<ing  problems 
which  members  assimilate  over  time.   Some  labs  are  noted  for  the  con- 
servatism of  their  designs;  others  are  gamblers  and  are  noted  for  far- 
out  thinking  and  occasional  outstanding  breakthroughs.  These  char- 
acteristics as  well  as  certain  of  the  long-run  organizational  goals 
become  engrained  in  the  members  of  the  organization.  This  indoctrination 
and  Its  consequences  are  described  in  the  following  manner  by  Simon 
(1957): 

The  organization  trains  and  indoctrinates  its  members.  This  might 
be  called  the  internalization  of  influence  because  it  Injects  into 
the  very  nervous  systemsof  the  organization  members  the  criteria 


TABLE  V 
USE  OF  OUTSIDE  SOURCES  OF  INFORMATION 


Mean  time  in  consulting  with  outside  sources:  11.6  man-hours 

Median  time  in  consulting  with  outside  sources:  0 

Range:  0-100  man-hours 

Number  of  proposal  teams:  116 

Number  of  proposal  competitions:  15 


Relation  of  time  spent  consulting  with  outside  sources 

Average  correlation 

(4) 

to: 

coefficient 

0. 

Technical  quality  of  proposal 

-0.30** 

2. 

Time  spent  In  literature  search 

0.31** 

3. 

Time  spent  consulting  with  specialists  within 
the  laboratory 

0.15 

5. 

Level  of  effort 

0.16 

6. 

Time  spent  in  analytic  design 

0.25* 

7. 

Time  spent  in  breadboard  ing  and  other  benchworl< 

0.15 

8. 

Proposal  team  size 

0.02 

9. 

Average  education  level  of  the  proposal  team*** 

-0.02 

10. 

Size  of  technical  staff 

-0.20 

1  1. 

Ratio  of  technical  staff  to  total  employment 

-0.24* 

Partial  correlations: 

Technical  quality  with  level  of  effort  constant 

-0.35 

<+4,0.5> 

Technical  quality^with  time  spent  in  analytic 

-0.28 

design  constant  (t.  -  ^) 
4,0,6 

Technical  Quality  with  size  of  technical  staff 

-0.24 

constant  (t4  g. I0^ 

Technical  quality  with  ratio  of  technical  staff 
to  total  employment  constant  (t      ) 

4,0.11 

-0.22 

Technical  quality  with  size  of  technical  staff 

0.18 

and  ratio  of  technical  staff  to  total  employ- 

ment  constant  (t^^Q  .q^,,) 

significant  at  0.05  level,  **  significant  at  0.001  level,  *"**  Ranl<  order  correlation. 
Refer  to  Table  11  for  the  method  of  ranl<ing. 

Correlation  coefficients  in  this  table  are  based  upon  data  from  the  15  proposal 
competitions  In  which  two  or  more  laboratories  reported  the  use  of  outside  sources 
of  information.  For  this  reason,  they  will  differ  from  their  counterparts  In  other 
tables  which  are  based  upon  all  22  competitions. 
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of  decision  that  the  organization  wishes  to  employ.  The  organi- 
zation member  acquires  knowledge,  skill  and  Identifications  or 
loyalties  that  enable  him  to  make  decisions,  by  himself  as  the 

organization  would  like  him  to  decide. 

The  preparation  of  a  proposal,  which  is  a  document  directed  to- 
ward outsiders  and  attempting  to  sell  the  organization's  point  of  view 
on  a  particular  matter,  will  almost  certainly  be  based  upon  these  in- 
ternalized decision  criteria.   In  order  to  make  use  of  the  contributions 
of  an  outsider,  such  implicit  rules  must  either  be  conveyed  to  him  or 
his  completed  work  must  be  re- interpreted  in  light  of  them.   What  this 
means  Is  that  a  greater  amount  of  information  must  be  transferred  when 
communication  takes  place  between  the  proposal  team  and  consultants 
who  are  outside  of  the  company  than  when  communication  is  with  con- 
sultants within  the  company.   In  the  latter  case,  the  decision  criteria 
are  stored  In  what  Simon  calls  the  organization's  memory  and  do  not 
have  to  be  made  explicit  in  the  communication  process. 

The  severe  time  constraint  under  which  proposals  are  written,  of 
course,  magnifies  the  problem.  The  inadequacy  of  outside  sources  may, 
to  Oi large  extent,  be  a  characteristic  of  the  proposal  preparation 
process.  The  period  allowed  In  the  cases  studied  ranged  from  30  days 
to  six  weeks.   If  channels  are  not  set  up  and  outsiders  prepared  as 
early  as  possible  (which  they  complain  is  often  the  case)  there  is  simply 
insufficient  time  to  use  them  effectively.   Because  the  dysfunctional  re- 
lation of  the  use  of  outside  information  sources  may  be  a  product  of  the 
time  constraint  under  which  proposals  are  prepared,  the  reader  should  be 
wary  In  extrapolating  these  results  to  R  &  D  projects  of  longer  duration. 
Evidence  from  the  data  and  from  interviews  with  project  managers  and  pro- 
fessional consultants  leads  to  the  recommendation  to  proposal  managers 
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that  they  attempt  to  predict  well  in  advance  the  areas  in  which  they 
will  need  help  on  a  proposal.  The  objective  is  to  provide  sufficient 
time  to  overcome  the  apparent  communication  problem,  so  that  they  may 
then  derive  greater  benefit  from  outside  sources  of  technical  in- 
format  ion. 

Use  of  outside  sources  is  more  strongly  related  to  the  amount  of 
time  spent  In  analytic  design  than  to  the  total  number  of  man-hours 
spent  in  preparation  of  the  profjosals.  Outsiders  are  utilized  in  those 
cases  in  which  a  greater  amount  of  analytic  work  is  done  during  the  pro- 
posal preparation  period.   Laboratories  which  spend  less  time  in  an- 
alytic work  during  the  proposal  preparation  period  are  those  which 
are  probably  more  competent  in  the  required  area  and  ^y   well  have  per- 
formed the  necessary  analysis  prior  to  the  receipt  of  the  RFP  (either 
through  pre-proposal  research  or  work  of  a  similar  nature  on  a  pre- 
vious contract)  or  are  simply  more  efficient.  On  the  other  hand,  firms 
spending  more  time  in  analysis  during  the  proposal  preparation  period 
may  be  trying  to  enter  a  new  business  area,  do  not  have  requisite 
experience,  did  less  work  in  the  area  prior  to  receipt  of  the  f^P  and 
have  to  rely,  at  least  temporarily,  upon  outside  competences.  The 
present  data  are  Insufficient  to  test  these  hypotheses,  but  it  can  be 
seen  that  outside  sources  are  of  little  help  in  improving  the  technical 
ranking  of  the  proposal. 

The  use  of  outside  consulting  Is  related  neither  to  the  size  of  the 
proposal  team  nor  to  its  level  of  education,  contrasting  with  what  was 
found  for  consulting  within  the  company.  The  use  of  outside  sources  Is 
then  not  dependent  upon  having  more  people  available  to  make  the  contacts 
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nor  does  It  appear  to  be  used  to  compensate  for  lack  of  education  on 
the  part  of  proposal  team  members.  Outside  consulting  Is  unrelated 
to  consulting  with  staff  specialists.   So  it  would  appear  that  the  two 
sources  are  consulted  for  different  reasons.  Staff  specialists  are 
probably  consulted  primarily  because  they  are  handy,  whereas  outside 
sources  are  resorted  to  only  when  specific  problems  arise. 

DISCUSSION 

The  failure  to  find  a  very  strong  relationship  between  use  of  any 
of  the  information  sources  and  rated  technical  quality  of  proposals 
Is  somewhat  surprising.   Expectations  relative  to  at  least  one  of  the 
sources  are  not  supported  by  the  evidence.  One  rather  important  qual- 
ifying question  remains  unanswered.  The  study  measured  expenditure  of 
effort  only  during  the  formal  proposal  preparation  period.   Work  during 
the  pre-proposal  period  Is  unknown.   We  do  know  from  interviews  with 
proposal  managers  that  it  Is  fairly  common  practice  for  at  least  some 
of  the  competing  labs  to  gain  an  early  start.  The  trouble  in  at- 
tempting to  measure  activity  prior  to  receipt  of  a  request  for  proposal 
stems  from  the  difficulty  in  determining  the  exact  starting  point.  This 
Is  often  not  clear  even  In  the  mind  of  the  proposal  manager.  Often  all 
of  the  activity  on  several  earlier  projects  could  just!  flably  be  ascribed 
to  preparation  for  a  given  proposal.  For  these  reasons,  the  survey  only 
attempted  to  measure  activity  performed  beyond  a  determinate  starting  point: 
the  date  of  receipt  of  the  government's  request  for  proposal.   Since  labs 
gaining  an  early  start  can  reasonably  be  expected  to  produce  higher  quality 
proposals,  the  net  effect  Is  probably  a  weakening  of  the  positive  cor- 
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relations  with  technical  quality. 

Of  the  Information  sources  considered,  literature  search  probably 
suffers  most  from  the  inability  to  measure  pre-proposal  activity.  A 
forthcoming  study  of  ongoing  R&D  projects  shows  that  literature,  in 
contrast  to  the  other  two  sources,  is  used  more  heavily  during  a  pro- 
ject's earliest  phases.   So,  if  pre-proposal  activity  could  be  properly 
measured.  It  might  be  expected  that  literature  search  would  show  some 
positive  correlation  with  quality.   In  addition,  the  correlation  between 
use  of  staff  specialists  and  quality  might  be  strengthened  and  the  in- 
verse relation  between  quality  and  the  use  of  outside  sources  might 
even  be  weakened. 

SUMMARY  AND  CONCLUSIONS 

Twenty-two  proposal  competitions  for  government  R&D  contract, 
involving  156  proposal  teams,  are  examined  to  determine  the  relative 
use  of  three  sources  of  technical  Information.  The  extent  to  which 
each  proposal  team  relied  upon  literature  search,  the  use  of  staff 
specialists  within  the  lab  and  the  use  of  outside  sources  of  Information 
Is  related  to  the  rated  technical  quality  of  Its  proposal,  and  to  other 
variables  characterizing  the  proposal  team  and  Its  parent  laboratory. 

Twenty-two  percent  of  the  total  time  expended  by  156  proposal  teams 
was  devoted  to  the  seeking  and  gathering  of  technical  Infonmatlon.  Of 
the  three  Information  sources  used,  only  one,  laboratbry  specialists, 
appears  to  be  at  all  directly  related  to  the  technical  quality  of  the 
product  and  this  relation  Is  weak  and  unreliable.  Technical  quality  Is 
Inversely  related  to  the  extent  to  which  the  proposal  team  relies  upon 
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Individuals  outside  of  the  laboratory  as  sources  of  Information.  This 
Is  partially  explained  as  a  result  of  the  lack  of  In-house  talent, 
with  the  use  of  outside  sources  representing  an  unsuccessful  attempt  to 
substitute  for  this  deficiency. 

Among  Information  sources,  the  use  of  literature  Is  directly  re- 
lated to  both  Internal  and  external  consulting  but  there  Is  no  relation 
between  the  two  types  of  consulting.  Most  of  the  proposal  teams  relied 
to  a  considerable  extent  upon  the  literature;  In  addition  they  consulted 
with  either  Internal  or  external  sources  of  expert  knowledge  or  both. 
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